From Cognitive Artifacts to Social Artifacts: The iDeas Design Ecology

This dissertation explores the integration of physical and digital interactions for artifacts used in design sketching and brainstorming. The proposed system supports design practice using an ecology approach; an ecology is a collection of tools and content used by a group such that the whole is greater than the sum of its parts. Expected contributions include novel models of sharing and collaboration; new designs for extending visibility via shared digital artifacts; lightweight methods for supporting design documentation; and evaluation of the effects of these approaches on design practice. We will evaluate these contributions by designing and implementing tools for browsing, sharing, and manipulating design content; testing components of these interfaces in laboratory studies; and assessing cumulative social and learning effects through longitudinal deployments. INTRODUCTION Early stages of design involve a significant amount of brainstorming and sketching. The primary tools used to structure these activities are paper notebooks and whiteboards. In design practice, notebooks and whiteboards are cognitive artifacts: external tools that help shape design cognition and collaboration. Designers’ use of notebooks and whiteboards is largely epistemic — the point of using them is not to produce, but to learn [8]. The emergence of augmented interfaces for integrating physical and digital interactions [4, 7, 9, 10, 13], combined with the search, annotation, and presentation mechanisms afforded by digital content, suggest new ways to support and augment design practice. This dissertation proposes a transformation of the cognitive artifacts of design into social artifacts, and examines the implications of such a transition. Thesis Statement and Contributions An ecology for design that integrates augmented physical and digital tools can facilitate collaboration between designers and improve the visibility of design practice. We will demonstrate this by implementing a working system and evaluating it in lab studies and longitudinal deployments. Specifically, we will perform evaluations of the following hypotheses (user experience goals in italics): H1 The lightweight digital capture of design artifacts in such an ecology can create incentives to author design content and decrease the effort needed to share and annotate it. (encouraging rich design documentation) H2 Interfaces that leverage the additional metadata afforded by augmented tools can ease the task of accessing design content of interest. (enhancing retrieval and review of design content) H3 An ecology approach to design tools, with an emphasis on novel mechanisms for browsing and sharing collections of content, can increase awareness of both one’s own work and the work of other designers. (promoting social translucence, reflection) This dissertation will also identify a conceptual framework and design guidelines for using integrated interactions in creative work, based on real-world observations of how designers make use of this new model of interaction. TOOLS, TECHNOLOGIES, PRACTICES Two primary artifacts for brainstorming and ideation are sketchbooks, also known as idea logs, and whiteboards (see Figure 2). In many ways, a physical notebook remains the best personal tool for authoring such content: it is excellent for sketching, it is lightweight, and its “display” has infinite battery life. Design notebooks provide a space for Copyright is held by the author. UIST ’06, October 15–18, 2006, Montreux, Switzerland. UIST 2006 Adjunct Proceedings: Doctoral Symposium 35 individual ideation and documentation: designers take notes, record meetings, and sketch and write down design observations in the field. Back at the studio and in meeting rooms, walls and whiteboards play an essential role for designers as a medium for individual and group ideation and presentation. Nevertheless, there are also limitations with physical media. Digital representations are much easier to copy, distribute, search, and store; and support remote and collocated collaboration. Electronic media now exist alongside physical media for individual ideation and documentation: laptops, cell phones, and digital cameras are ubiquitous. Interactive prototypes and written documents are created on the computer, often based on notes. Large electronic displays allow designers to present and author digital content in group settings. ECOLOGIES FOR DESIGN To address and embrace this plurality of artifacts within design practice, we are developing the iDeas design ecology. The iDeas design ecology integrates physical and digital tools into a social system which supports and augments design through the sharing of design artifacts (see Figure 1). We discuss three salient features of the iDeas ecology: capture and access, sharing, and annotation. Capture and Access In the iDeas design ecology, the physical tools that designers currently use — notebooks and walls — are not replaced, but rather augmented. Augmented physical tools take advantage of the familiar, nuanced interactions available with physical artifacts while imbuing them with affordances of electronic media [10, 14]. The iDeas ecology supports the capture of common physical writing surfaces. Handwritten notes are captured using the Anoto digital pen system [1], which has a normal ballpoint pen tip for writing and is also augmented with an embedded camera for digitizing what is written. Physical whiteboards are captured using a dedicated digital camera; digital whiteboard captures are saved as digital images. Designers may document ethnographic studies using digital cameras, take quick snapshots using camera phones, jot down notes in a text file, or find inspiration in material downloaded from the web; any digital image or document may be directly imported into the iDeas system. Captured content is indexed and stored either in the design repository (notes text) or online through a service that is integrated into the iDeas ecology via mash-up programming (images). The primary access component of the iDeas ecology is the ButterflyNet browser [17], which offers a rich desktop interface for visualizing and manipulating captured design activity. A context panel automatically presents data related to the items in focus (e.g., images taken around the time the page was written). Other panels display visualizations of content, such as a timeline visualization that displays calendar milestones and allows users to jump to notebook contents by date. Sharing and Annotation One powerful affordance of augmented physical tools and digital tools is rich, ubiquitous digital capture. In addition to digitally capturing the content itself, these tools capture metadata: timestamp, location, and other contextual properties not accessible from traditional physical tools. Both physical and digital tools in the iDeas system offer new methods for access and manipulation of captured content, including sharing, search, and annotation. Ubiquitous digital capture enables easy sharing of media, as well as simple searching of content by timestamp and other metadata. Recent years have also seen the rise of the folksonomy [16]: collaborative keyword categorization by users of an information system. In the iDeas ecology, designers may annotate content using similar techniques. Any content — such as notebook pages and photographs — may be labeled with a tag (classification keyword) or a flag (attention mark) to create a set of information; these sets may then be searched or shared. Tags and flags are simple, effective mechanisms for finding, documenting, and understanding content belonging to oneself or others. DESIGNING FOR DESIGN As part of our background research into how integrated interactions might influence the culture and practice of design, we conducted two longitudinal studies of design technology and practice in the context of design education [11]. Across two quarter-long studies, fifty-eight students used early versions of the iDeas design ecology, authoring more than 4,000 pages of content in the course of their work. We also examined current design practices by observing group meetings and conducting interviews. These studies of design education have produced several insights. In the following sections, we identify three facets of design practice that demonstrate the potential of a social approach to cognitive design artifacts. Figure 2. Design in situ. Designers and students work with a variety of physical and digital media during group activities. 36 UIST 2006 Adjunct Proceedings: Doctoral Symposium Collaboration In almost all projects, whether in schools, in design firms, or elsewhere in the “real world,” creative work is usually done in teams. Design projects feature strong interplay between times of individual ideation and reflection, and times of group presentation, discussion, and brainstorming. One model we are exploring for explicit sharing is the group notebook. Conceptually, group notebooks are shared repositories, similar in concept to text-based Wikis but incorporating sketches and other media. Group notebooks can be used to share design content with other group members and project mentors, or to bookmark and preserve important data for later retrieval. Designers may place content from their personal notebooks, whiteboards, or any other sources (e.g., links, text, documents, presentations, schematics) into the shared space. This pasting may be done either by physical gesture for captured writing surfaces such as notebooks or walls, or by digital selection and tagging at the desktop in the ButterflyNet browser. Later, group members may review the contents of the group notebook through the browser. The digital nature of the notebook allows users to add hyperlinks and to view content in a number of ways: sorting or filtering by date, by contributor, by tags, etc. Visibility Ecologies for design also enable more implicit forms of collaboration. In this research, we are exploring the concept of visibility: the extent to which the activities of a practice are made visible to colleagues and onlookers through the artifacts and performance of the activity. One prominent theme is the potential for making design activity more visible through shared digital representations. Much as the cluttered studio makes design more visible to observers through the display of physical artifacts, peripheral presentations of electronic content can make design more visible through the display of digital artifacts. Inspired by the exploratory views of social software websites, the iDeas design ecology will provide ambient awareness of design activity by proactively displaying visualizations of design content (see Figure 3, left). When not in direct use, iDeas desktop displays and digital whiteboards will present snapshots of current activities, such as newly uploaded photographs, whiteboard captures from recent brainstorming sessions, and the latest notebook entries from colleagues. Digital visibility poses broad questions for both content authors and observers. For authors, presentation of self becomes a concern when all content is potentially visible to others; for observers, the expected cost of information is a consideration when confronted with lots of visualizations. We are designing algorithms for adaptively displaying content based on relevance and context, and will apply them to both ambient displays and interactive browsers. Reflection The process of reflection is an integral part of the design experience. Reflection helps designers further crystallize their understanding of experiences, making the intuitive knowledge therein more easily shared with others. By emphasizing ease of documentation, the iDeas design ecology facilitates reflective practice through composition of presentation documents such as electronic portfolios [3]. A common shortcoming in design reflection is the inability to easily reference design rationale: the reasoning behind the development of a design artifact [12]. Even when extensive documentation exists, e.g., in a designer’s paper notebook, finding the exact page of notes that explains why a certain decision was made can be tedious. While there have been many efforts to preserve design rationale, most of these have not succeeded due to the high immediate costs relative to uncertain future benefits. We believe that the iDeas design ecology, with its informal approach to annotation that emphasizes fluidity and minimizes overhead, will encourage design documentation and enhance design rationale retrieval. Designers can use the standard tagging mechanisms to indicate design content relevant to a given project. We will also explore the use of specialized ink gestures to indicate more formal design rationale relationships in written notes (see Figure 3, right). EVALUATION In general, we will take a two-pronged approach to evaluation. Lab studies will be used to evaluate individual components of the iDeas system where long-term social effects are less relevant, e.g., the graphical user interfaces for browsing group notebooks or physical gestures for applying design rationale labels. We will also continue to conduct longitudinal studies of the iDeas ecology. We will deploy iDeas to undergraduate and graduate students enrolled in design classes lasting 10 to 20 weeks, with students generally working in groups of three to five. Data collection will include captured design content, user interaction logs, meeting video, performance data, questionnaires, and exit interviews. We believe that longitudinal studies are valuable for assessing the ongoing relationships and long-term effects of collaborative technologies. The iDeas ecology allows us to instrument design activities and observe how practices actually change over time. Nevertheless, this research presents interesting challenges for evaluation: how to Figure 3. Left: Prototype of the iDeas proactive media display, featuring snapshots of team and organizational activity. Right: Prototype of a design notebook with tags and symbols for design rationale. UIST 2006 Adjunct Proceedings: Doctoral Symposium 37 meaningfully assess such abstract concepts as collaboration and visibility is an open question. To evaluate hypothesis H1, we plan to observe longitudinal deployments, measure the amount of content captured, shared, and annotated, and compare activity in the iDeas system with current practice (students using physical notebooks and other commonly available tools). To evaluate hypothesis H2, we plan to conduct laboratory studies of our adaptive presentation algorithms, measure performance on browsing and search tasks, and compare the iDeas interfaces with traditional browsing interfaces over the same data sets. We will also measure longitudinal use of our browsers, as we expect users to reap additional benefits from familiarity with content and annotations. To evaluate hypothesis H3, we plan to deploy iDeas tools and awareness displays in longitudinal settings, and attempt to elicit and measure the awareness effects of our system through observation, questionnaires, and analysis. Furthermore, we believe that iDeas will have beneficial effects not only on the user experience of designers, but as a tool for supporting learning, teaching, and research. Maldonado, a collaborator on the iDeas project, will evaluate the effects of the ecology on learning and educational processes as part of her dissertation work. RELATED WORK A number of systems have inspired various elements of our work. Mackay’s early work with augmented paper (e.g., [10]) demonstrated the importance of leveraging current physical practices. NotePals [4] first introduced the idea of shared electronic repositories for paper-based notebooks. Collaborage [13], Liveboard [5], and Outpost [9] examined the use of physical and digital whiteboards for collaboration. MessyDesk and MessyBoard [6] explored networked sharing of content on desktops and whiteboards. The eClass project [2] studied automatic capture and access of classroom whiteboard materials. Several systems have looked at the use of device ensembles for capture and access, including ButterflyNet [17] and Books with Voices [7]. INCA [15] is a toolkit for capture and access that influenced the architectural design of the iDeas ecology. CONCLUSIONWe have presented the iDeas design ecology, a systemwhich integrates physical and digital tools in support ofdesign practice. Our earlier studies of the iDeas systemhighlight the potential for augmenting design practicethrough the sharing of design artifacts. We are continuingto build components for iDeas, and look forward to furtherexploring the design space of ecologies for creative work. ACKNOWLEDGEMENTSI thank my advisors, Scott Klemmer and Terry Winograd,for their encouragement and guidance in shaping this work;Heidy Maldonado, for her collaboration on the iDeasproject; and Lora Oehlberg and Mike Krieger, for theirinsightful discussions. I also thank the Wallenberg GlobalLearning Network for supporting this research.REFERENCES1 Anoto AB, Anoto Technology. http://www.anoto.com 2 Brotherton, J. A. and G. D. Abowd. Lessons learned from eClass: Assessing automated capture and access in the classroom. ACM Transactions on Computer-Human Interaction (TOCHI) 11(2): ACM Press. pp. 121–55, 2004. 3 Cambridge, B. L., ed. Electronic Portfolios: Emerging Practices In Student, Faculty, And Institutional Learning.